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Introduction 



This report is based on an almost 5-year 
continual study of selected collaborates 
engaged in education reform and improvement 
with a focus on science, math, and technology 
in schools serving K-12 students in rural 
America. (Formal annual and end-of- 
project/final reports have been submitted to 
the National Science Foundation, and more 
than 20 other reports of specific investigations 
or substudies have been produced and are 
available for public review.) Each 
collaborative included in this study was 
funded by the National Science Foundation 
(NSF) as a part of its Systemic Reform 
Initiative (SRI) program. 

This version of the final report of the study is 
primarily intended for use by educators, 
decision makers, and others who are interested 
in education program improvement from the 
perspective of the local school and 
community. This report does not reflect all 
elements of this study, nor does it attempt to 
evaluate the NSF or the Rural Systemic 
Initiative (RSI) of this government 
organization. It is an attempt to describe some 
of the critical components of the effort, the 
perceptions of these at the local level, and an 
objective view of the impact and change that 
occurred. 

The format and style of this report were 
selected to provide an easily readable and 
nontechnical description of selected 
communities and to encourage a critical 
comparison of these with those of the readers 
who may be involved in systemic or other 
large-scale school improvement efforts. 
Further, it provides summarized feedback 
from a variety of local project participants and 



other stakeholders on their views and values 
of critically identified “drivers” of education 
reform and their perceptions of efforts to 
improve/reform local education programs 
through large-scale initiatives, as designed by 
collaboratives with federal funding. 

In response to a “Dear Colleague” letter from 
the National Science Foundation that 
requested proposals to conduct studies of 
rural, urban, and state systemic reform 
initiatives, researchers from a number of 
universities and other organizations submitted 
requests for support of a variety of studies. 
None of the proposed studies focused on the 
Rural Systemic Initiative, and most proposed 
approaches using state-based test scores and 
the intentions of the projects (collaboratives) 
as indicators and guidelines of the studies. 
Having need for a study of all three major 
components (rural, urban, and state) of the 
systemic component of the NSF program, 
representatives of the National Science 
Foundation requested a proposal from Dr. 
Jerry Horn, Principal Research Associate at 
The Evaluation Center at Western Michigan 
University (WMU), to study the Rural 
Systemic Initiative. In the discussion phase 
that followed, Dr. Horn explained that his 
proposal would focus on the local perspective 
and that it might be quite different from others 
who would favor collecting large amounts of 
collaborative project data and major use and 
reanalysis of state test scores or other 
standardized test data sets. At the same time, 
NSF representatives indicated that a proposal 
from Dr. Horn would be subjected to the same 
scrutiny and evaluation of any other proposal 
submitted to the Foundation. With agreement 
on all of these elements, Dr. Horn and his 
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associate at The Evaluation Center, Dr. Craig 
Russon, developed a proposal and submitted 
it to NSF for consideration. After the required 
review and evaluation, NSF awarded a three- 
year grant to The Evaluation Center for “The 
Rural Systemic Initiatives Evaluation Study,” 
with Dr. Horn as Principal Investigator. 
Operationally, the project initially was 
planned to be conducted largely from 1999 to 
2001. Three currently funded collaboratives 
were selected to be the focus of the study 
because it was thought that these were made 
up of school districts serving the traditional 
rural communities. Those selected were the 
Appalachian Rural Systemic Initiative (ARSI), 
the Delta Rural Systemic Initiative (Delta 
RSI), and the UCAN Rural Systemic Initiative 
(UCAN RSI). ARSI included school districts 
in the Appalachian area of Kentucky, North 
Carolina, Ohio, Tennessee, Virginia, and West 
Virginia; the Delta RSI had participating 
school districts from the delta region of 
Mississippi, Louisiana, and Arkansas; and the 
UCAN RSI focused on selected school 
districts in Utah, Colorado, Arizona, and New 
Mexico. 

In Spring 2000, Dr. Horn and The Evaluation 
Center were asked to consider an expansion of 
the study to include 3 additional collaboratives 
that had been funded recently or were 
scheduled for funding in the near future, i.e., 
the Texas RSI; the Michigan RSI; and the 
Coastal RSI, with school districts in Virginia, 
North Carolina, and South Carolina. 
Accepting this addition and challenge to the 
study, the project was extended, with an 
expected completion date of May 31, 2003. 
Each collaborative will be described in more 
detail later in this report. Over time, more 
than 15 RSI collaboratives were funded by the 
NSF, but only the 6 identified above were a 
part of this study. 



For many years, the NSF, other federal 
agencies, state governments, and other public 
and private entities, including local schools, 
have focused on education reform and school 
improvement. Some of these efforts 
supported a variety of change mechanisms, 
but in the mid-1990s systemic reform was in 
the vogue. For major and long-lasting change 
to occur, it was thought that the impact must 
be continuous throughout the grade levels (K- 
12) and must (1) include all schools within a 
system, (2) relate to alignment of the 
curriculum with recognizable standards, (3) 
include policies that relate to these standards 
and goals, (4) be reflected in budgets, and (5) 
be a part of the ongoing educational 
experiences of all children. Since math, 
science, and technology are the recognized 
interests of NSF, these became the focal 
points of the systemic initiative. Earlier 
efforts by NSF included summer and 
academic year institutes and workshops for 
teachers, the development of curricular 
materials, and a variety of other programs that 
identified teachers or a single subject area as 
the target for improvement. 

Systemic reform had a broader connotation 
and one that seemed to be well received by K- 
12 schools and higher education scientists, 
mathematicians, and science and math 
educators, who likely would be providers of 
services and knowledge for these projects. 
Also, state departments of education had some 
experience in this approach in earlier state 
systemic initiatives. Yet, NSF and other 
planners had little existing experience for 
building large-scale collaboratives (in terms of 
number of participating units and geographic 
area) that differed considerably from the 
earlier conceived and implemented urban Sis. 
These earlier Sis were single districts with 
limited geographic areas to be served when 
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compared with the rural systemics. While 
there were some similarities with the state 
initiatives, the RSIs often involved multiple 
states, regulations, and governmental 
jurisdictions. Whether anticipated or not, the 
RSIs faced some interesting, if not even 
major, challenges to fulfill the expectations of 
the NSF program as well as what was 
promised in the collaboratives’ proposals. 
Further, rural schools and the communities 
they serve are radically different from 
suburban and urban school districts. While 
this topic will be addressed in more detail 
later, suffice it to say that rural communities 
have a distinct culture that is based on trust, a 
sense of ownership and pride in their schools, 
and caution with regard to external 
interventions in their schools. Overall, the 
RSI program faced some existing obstacles 
from the outset. 

The principal investigator of this evaluative 
study, Dr. Horn, was particularly interested in 
this study because of his background and 
experience in science education and rural 
education. He had served as the national 
president for the Association for the Education 
of Teachers in Science (AETS) and the 



National Rural Education Association 
(NREA), as well as serving on the Board of 
Directors of the National Science Teachers 
Association (NSTA). Much of his research as 
professor at the University of South Dakota 
and Kansas State University related to the 
improvement of science in schools and rural 
education. He founded the first officially 
recognized “Rural Research Center” under the 
auspices of the NREA at Kansas State 
University in the early 1980s. After joining 
The Evaluation Center at Western Michigan 
University in 1994, he led the evaluation of a 
number of educational initiatives, including 
the charter school programs in Michigan and 
Connecticut. 

Even with this strong base of experience and 
knowledge of science education and rural 
education, he broadened the base of expertise 
for this project with the creation of a Research 
Advisory Team, which included the careful 
selection of individuals who are nationally 
known and who bring additional credibility to 
this study. The membership of this team and 
some of their involvement will be discussed 
later in this report. 
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The Rural Systemic Initiative of the National Science Foundation 



An overview of the Rural Systemic Initiatives 
program, as taken from the official Web site 
<http://www.ehr.nsf.org/hrd/rsi.asp> for this 
program, is as follows. 

The Rural Systemic Initiatives in Science, 
Mathematics, and Technology Education 
Program was developed in FY 1994. RSI, like 
the Urban Systemic Initiatives and the 
Statewide Systemic Initiatives, stimulates 
system-wide educational reform of science, 
mathematics, and technology. 

RSI is focused on improved education for 
students in rural, economically disadvantaged 
regions of the nation, particularly those that 
have been underserved by NSF programs; and 
on sustaining the improvements through 
encouraging community participation in 
instructional and policy reform. RSI is 
tailored to address policy, leadership, and 
workforce issues related to education, to 
provide a comprehensive and sustainable 
framework for science, mathematics, and 
technology education technology in 
elementary, secondary, and higher education. 

RSI targets regions that are highly rural; 
characterized by significant levels of poverty 
among their school-age children; and that 
share common cultural, social, and economic 
characteristics. Interested individuals or 
identified leaders within eligible regions form 
consortia that include large NSF -funded 
Initiatives (e.g., SSI, EPSCoR, USP). RSI 
regions can be geographically vast, typically 
crossing state lines, or may be composed of 
areas that are geographically separated but 



linked by a unitary vision and other 
commonalities. 

The RSI project scope is divided into two 
strategic elements. The initial element is a 
Development phase, to support planning for 
structuring Implementation ( second ) phase 
vision, strategies, and priorities. During the 
Development period, projects conduct 
regional self-studies to gather pertinent 
information regarding target populations, 
regional strengths and barriers, and policies 
that enhance or hinder instructional reform. 
Moreover, successful Development projects 
should result in a viable evaluation strategy to 
be utilized during the Implementation phase. 
The second strategic element, the 
Implementation phase, focuses on realizing 
the strategies for systemic improvement in RSI 
districts, schools and classrooms. The 
strategies can include teaching workforce 
enhancements, curriculum innovation, 
leadership development among teachers and 
local district administrators, and innovative 
and pertinent assessment strategies; and must 
result in better classroom instruction and 
higher student achievement. 

RSI goals include: 

The improvement of science, 
mathematics, and technology 
education in rural, economically 
disadvantaged regions of the nation. 

The preparation of a technologically 
competent workforce to enhance the 
infrastructure of economic 
development activities within a 
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community or region, by strengthening 
the science, mathematics, and 
technology instructional capacities of 
regional colleges and universities. 

The enhancement of scientific literacy 
and science understanding and 
appreciation among students and the 
general community in rural, 
economically disadvantaged regions 
of the nation. 

The development of community 
infrastructure to provide resources to 
sustain educational improvements. 

RSI collaboratives developed in a number of 
ways and through the efforts of individuals 
and groups. In some cases, an individual 
served as the initiator of the idea and he/she 
was able to put together a consortium of 
individuals/schools with a common interest 
and set of goals. In other cases, earlier efforts 
for science and math education had produced 
workable groups that wanted to continue this 
relationship and effort with new and 
potentially a major increase in support. In the 
application materials, development grants 
were made available by NSF on a competitive 
basis and groups were encouraged to pursue 
this opportunity as means to develop the 
collaborative and the design of a full plan for 
the systemic improvement project. (In this 
report, we refer to a program as the overall 
entity at the NSF level and a project as one 
funded element at the implementation level, 
such as a collaborative.) 

It should be understood that while this 
program was entitled “rural,” it had specific 
qualifications that excluded major geographic 
areas of what is commonly known as rural 
America, e.g., those areas that did not meet 



the criterion for “highly rural” and 
“characterized by significant levels of poverty 
among their school-age children.” In the 
application material, eligible school districts 
are defined as “those designated as ‘rural’ or 
‘small town’ according to the U.S. 
Department of Education, National Center for 
Education Statistics, and in which greater than 
30% of the school-age children are living in 
poverty.” 

There are many definitions or 
characterizations of “rural” and particularly 
“rural schools.” However, the common 
elements of these characterizations include 
considerations of sparse populations and 
isolation (from major residential and 
commercial areas), which might be the result 
of geographic distance or geographic barriers, 
i.e., mountain ranges, water expanses, absence 
of passable roads during seasonal periods, etc. 
However, there are other, more subtle barriers 
to access to metropolitan areas, such as 
culture, religious or language differences, 
socioeconomic status, historical events, etc. 

According to official U.S. Census Bureau 
definitions, rural areas comprise open country 
and settlements with fewer than 2,500 
residents < www.ers.usda.gov/Briefing/ 
Ruralitv/What is Rural)> . Urban areas 
comprise larger places and densely settled 
areas around them. Urban areas do not 
necessarily follow municipal boundaries. 
Most counties, whether metropolitan or 
nonmetropolitan, contain a combination of 
urban and rural populations, and since 
metropolitan and nonmetropolitan status is 
determined on the basis of counties, a rural 
school within a defined metropolitan area was 
ineligible to participate in an RSI 
collaborative. 
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The Beale Codes are used by the U.S. 
Department of Agriculture to define points on 
a continuum from 0 (most metropolitan) to 9 
(most rural). A designation of 8 (no places 
with a population of 2,500 or more and 
adjacent to a metropolitan area) or 9 (no 
places with a population of 2,500 or more and 
not adjacent to a metropolitan area) would be 
“highly rural.” When queried on this issue, 
RSI consortium leaders often referred to the 
Beale Codes as the indicator they used to 
determine which school districts would be 
invited to participate in the projects. This is 
an important point, because it helped 
determine participation and guided this 
study’s selection of schools with whom to 
compare selected RSI schools’ achievement 
on substudies described later. 

The use of governmental definitions of rural 
is further complicated by the fact that many 
school districts are “county school districts,” 
which may contain some schools that serve 
urban children and others that serve rural 
students. In rural education circles, it is 
commonly stated that about two-thirds of the 
schools in this country are rural and one- 
fourth to one-third of the students attend a 
rural school. Further, rural is a state of mind 
in that there is a sense of value for the land, 
the culture, and the rural life style. Others 
may have an address and a home in a rural 
area but work and have a social life with 
suburban or urbanites, but they do not share 
the values and way of life of the traditional 
rural resident. Expectations of students and 
schools and the value placed on specific 
factors may differ drastically between the 
traditional rural family and the newly arrived 
suburban or urban family with a residence in 
a rural area. 



With regard to the requirement that 
participating schools must have “significant 
levels of poverty among their school-age 
children,” there is the U.S. Census definition 
of poverty, which is based on income and the 
size of the family unit, and then there is the 
U.S. Department of Education’s common 
usage of “eligibility for free and reduced 
lunch.” While these definitions use similar 
data, eligibility for free and reduced lunch is 
usually determined only when a student or 
his/her parent/guardian requests consideration 
for this status. School people routinely will 
tell you that publicly reported data for free and 
reduced lunch for high school students are 
substantially underestimated. In many rural 
areas, there is a stigma associated with being 
tagged as living in poverty. 

The third requirement for participation is 
related to a sharing of common cultural, 
social, and economic characteristics, which 
seems to be largely included to accommodate 
collaboratives that would serve Native 
American children. At the same time, a 
special section of the solicitation notice NSF 
01-57 provides for a “Tribal Colleges and 
Universities Component.” 

This information related to requirements for 
participation is included in this report, not to 
infer that any school districts in the 
collaboratives were ineligible, but to indicate 
that this is not a program that is inclusive of 
all rural schools or schools serving rural areas. 
Further, it was quite likely that the 
requirement that 30 percent of the students 
must be living in poverty may not be a true 
estimate of reality. In summary, by definition 
entire states would be excluded from 
participation in the RSI program, even though 
they may have considerable geographic area 
that is rural. It seems most likely that the RSI 
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program was conceived and planned to serve 
a fairly narrowly defined group of schools and 
children. Thus the title of the program, Rural 
Systemic Initiative, may not be a totally 
accurate descriptor. 

Early descriptions of the program, including 
the program solicitation document (NSF 01- 
57), refer to “six impoverished regions,” but 
they are not named. Later in that same 
document, it says that by 1994 “the program 
made four implementation awards to six of 
these sites.” Thus, it would seem that the 
areas served by the ARSI, Delta RSI, UCAN 
RSI, and the Alaska RSI were a part of the six 
predetermined areas to be targeted by this 
program. By early 1999, the program had 
made implementation or development awards 



to 1 1 regions. It is not clear whether these 
grants were made in the “six impoverished 
regions” or whether the target area for the 
program had been expanded from 6 to 11. 

Funding for the implementation stage was 
restricted to those collaboratives or consortia 
that had received Development Awards. 
Implementation awards were expected to be 
funded at a level of $500,000-$ 1 .5 million per 
year, depending on the size of the consortium, 
not to exceed $6 million in 5 years. Further, 
during their fifth year there was an 
opportunity to apply for funding beyond the 
initial 5 -year implementation period in an 
amount up to $1.5 million per year, not to 
exceed $6 million in 5 years. 
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Rural Education and Schools Serving Rural Areas 



While there are a variety of definitions of 
rural, we can reasonably say that two-thirds of 
the schools and approximately one-fourth of 
the students who attend public schools reside 
in rural areas of this country. Prior to the mid- 
1970s, rural education or schools with 
enrollments of less than 300 were not 
recognized in many of the statistical reports 
and there was little interest in what they were 
doing. The concept of a “one best model” for 
schools existed, and the quickest way to get to 
that model was to conduct massive 
consolidations. However, a nucleus of 
researchers began to look seriously at the data 
and discovered that some of these small 
schools were the most effective schools that 
existed. Largely, the criticisms of the small 
schools related to a “restricted curriculum” 
and inefficiency, i.e., operating costs per 
student. 

Without going into lengthy discussions of the 
pros and cons of supporting schools with 
small enrollments, there has been a rather 
substantial movement to rural areas by people 
who maintain employment in suburban and 
urban environments. Oftentimes, these new 
residents are well-educated professionals who 
simply want to live in a quieter and less 
congested environment. This type of rural 
community is quite different than the 
thousands of villages and towns across the 
country that have served rural residents for 
generations. Their schools are the heart and 
pride of their communities, and they challenge 
efforts to consolidate with another community 
that has been their rival in school sports for 
years. 



Today, some rural communities protect 
themselves from suburban sprawl by 
restricting the number of acres that can be sold 
from farmlands for housing. Residents value 
their lifestyle and the institutions of their 
communities, including the local schools. 
Where schools have been closed for whatever 
reason, this has triggered the decline of the 
community as a social institution and as a 
business factor for the residents. Large 
discount stores have forced the closure of 
family-owned businesses, and government 
agricultural programs have on occasion 
“emptied” rural communities of many of its 
residents. Coupled with the lack of 

employment for young people, rural 
communities that are far from large cities or 
major transportation routes have declined and 
will likely continue to do so in the future. 

The RSI program was designed to address the 
science and math needs of these areas, i.e., 
those that are “highly rural and poor.” Some 
rural families have considerable wealth in the 
form of land and equipment, but they are 
income poor. A common pattern is for one or 
more adult members of farm families to work 
off the farm. As a result, farming is done in 
the evenings and on weekends, which leaves 
little time for these hardworking people to 
share time with their families and especially 
their children. Plus, much time to support the 
school or become involved in school-related 
work is not available. 

Schools in rural America range from a few 
remaining one-room schools to large schools 
with several thousand students that serve an 
entire county. The smaller the school and the 
more geographically isolated it is, the more 
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difficult it is to attract well-qualified teachers, 
especially math and science teachers. 
Oftentimes, provisional certification or 
emergency is the route followed to address the 
unavailability of teachers in these areas. 
Also, there may be only one high school 
science teacher, and if he/she goes, the 
program goes as well. Thus, there is a real 
need for a written curriculum that teachers 
follow to ensure continuity for students. 

However, it is interesting that charter schools 
and private schools have been developed in 
the interest of some of our more privileged 
parents, and they have adopted many of the 
characteristics of small schools serving rural 
areas, i.e., small class size, individualized or 
small group instruction, parental involvement, 
community-based instruction, looping 
arrangements for assignment of teachers, etc. 

In the last 25 years, rural education has 
become a much more prominent fixture in 
education research. The National Rural 
Education Association (NREA) has grown 
from a semisocial/professional affiliate of 
county school administrators to a 
sophisticated body of professionals from K- 1 2 
schools, higher education, government 
agencies, and business and industry. It has a 
research component in which research is 
encouraged and recognized in various ways. 
A number of rural education centers around 
the country were developed to serve as 
advocates, provide needed services, and 
conduct and coordinate research appropriate 
for small schools in their service areas. The 
ten nationally funded regional research centers 
across the U.S. have worked under directives 



that a specific amount of their programmatic 
resources (25%) must be allocated to rural 
schools and their needs. 

In summary, rural education is as diverse as is 
rural America. Some schools are financially 
well supported with the most up-to-date 
technology supporting some of the most 
creative instruction and curriculum you can 
find. On the other hand, there are schools 
serving rural areas that are without adequate 
resources and incapable of much 
improvement. Yet, there is the entire range of 
schools with varying capacities for offering 
students quality programs in math and science 
that could launch them into promising 
professional careers. Some limitations or 
potential barriers for these students are 
financial, and some are due to lack of 
opportunities. It is in the interest of this 
country and it is a right of all students to have 
an opportunity to develop to the fullest of their 
academic capabilities. Some schools within 
the realm of rural education, just as some in 
urban and suburban areas, meet these 
expectations. 

In 1995, the Association for the Education of 
Science Teachers (AEA) devoted its yearbook 
to “rural science education.” A number of 
other papers and research reports related to 
rural education can be found in The Rural 
Educator, the official publication of the 
NREA. For those seeking further information 
regarding rural education, various reports and 
writing by such authors as Paul Nachtigal, 
Jonathon Sher, Jerry Horn, Robert Stephens, 
and Toni Haas are recommended. 
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The Drivers of Education Reform 



While the goals of the RSI program are 
prominently stated, a set of six elements of 
achievement toward systemic reform were 
identified and disseminated by NSF. These 
elements became known as the “Drivers of 
Systemic Reform,” and they became the focal 
point for much of the planning, operations, 
and planning of the RSI program. The 
drivers, as developed and promoted by NSF, 
are listed below. 

# Implementation of a comprehensive, 
standards-based curriculum and/or 
instructional materials aligned with 
instruction and assessment, available 
to every student served by the system 
and its partners 

# Development of a coherent, consistent 
set of policies that supports provision 
of high-quality mathematics and 
science education for each 
student/excellent preparation, 
continuing education, and support for 
mathematics and science teachers 
(especially of the elementary level); 
and support for administrators who 
have responsibility for implementing 
science and mathematics education 
reform 

# Convergence of all resources that are 
designed for or that reasonably could 
be and to support science and 
mathematics education — fiscal, 
intellectual, and material — both in 
formal and informal education 
settings — into a focused program that 
upgrades and continually improves the 



educational programs in mathematics 
and science for all students 

# Broad-based support from parents, 
policymakers, institutions of higher 
education, business and industry, 
foundations, and other segments of the 
community for the goals and collective 
value of the initiative 

# Clear evidence that the program is 
significantly enhancing student 
achievement and participation in 
science and mathematics through a set 
of indices that might include 
achievement in standard and 
performance-based tests, portfolio 
assessments, course enrollments, 
college admission rates, higher level 
courses passed, advanced-placement 
tests taken, perceptions of local 
employers, and college majors in 
SMET 

# Significant reductions in the 
achievement of disparities among 
students that can be attributed to 
socioeconomic status, race, ethnicity, 
gender, or learning styles 

Because of the nature of rural schools/- 
communities and the timing of this evaluative 
study (late 1998 and early 1999 with an 
expansion in 2000), a number of issues 
impacted the study. For the sake of 
understanding them, I have related them 
directly to the most appropriate driver. 

Almost every state initiated actions to create 
and implement standards and benchmarks 
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with an accompanying set of mandated 
statewide assessment instruments. Thus, 
action to align curriculum with state tests 
occurred. Were actions to align the science 
and math curricula a mutually supportive 
endeavor (NSF programs and state 
requirement) or a timely coincidence? 
Further, the No Child Left Behind (NSLB) 
initiative by the federal government had late 
impact on this area of reform. However, the 
consortia that served as the focus of this 
evaluative study occurred before NCLB 
became a significant consideration. 

Policies in many small/rural communities are 
patterned after a generalized set often 
promulgated by the state association of school 
boards. In effect, a set of policies that cover 
the required elements of the operation of a 
school district serve as the basis for some 
customizing to meet local demands. Further, 
the specific aspects of “schooling” that served 
as the focus of the RSI program are not 
addressed in school district policies. 
However, a number of practices are 
commonly known and understood, and in 
effect these serve the purpose of written 
policies. In rural communities, informality is 
more often the manner in which education 
programs operate. For example, the science 
teacher for the high school might be given the 
responsibility for reviewing textbook options 
without there being a specific policy as to how 
that will occur. In a larger and more urban 
setting, there is likely a specific set of policies 
that deal with the manner in which review 
committee members are selected and how 
such a review will be conducted and 
considered by the local school board. 

Resources in many school districts are scarce 
and hardly of such magnitude that major shifts 
can occur. However, simply defining that 



library materials or software will first be used 
to address science or math needs would be a 
positive indicator that this driver is being 
seriously considered. Financial resources 
really became the focus of this driver, even 
though it is much broader than this, including 
a call for converging intellectual capital into 
this effort. 

Broad-based support (from parents, 
policymakers, etc.) has a variety of operational 
interpretations. Something as noncommittal 
as attendance at planned events or willingness 
to be listed as a member of an advisory 
committee or board can be construed as 
evidence of broad-based support. However, 
this author has a much higher standard for 
“broad-based” support, which would include 
making substantial commitments of financial 
resources for professional development for 
science/math teachers, assignment of 
personnel by a university or company or 
willingness of an individual to work with a 
science or math teacher over a long period of 
time to develop inquiry laboratory activities, 
parents who are willing to commit several 
hours a week to a specific task related to 
improving science and math experiences for 
children and youth, or a community group’s 
efforts to raise substantial amounts of money 
to purchase up-to-date technology for the 
school, etc. 

Clear evidence that the program is enhancing 
student achievement is difficult to discern, 
since there are any number of ongoing 
interventions in schools. To attribute success 
(or failure) to any one of them is a bit of a 
stretch of both logic and good research. The 
types of achievement included with the driver 
are meaningful and observable but, 
unfortunately, the tenor of the times focuses 
on only one measure of achievement — the 
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results of state-mandated tests. Since most of 
the RSI consortia are multistate projects, it is 
difficult to interpret the various reporting 
systems of the states and/or provide clear 
evidence that the test reflects the instruction 
to which students have been exposed. 
Certainly, the alignment of the curriculum 
with a common set of standards helps alleviate 
this problem. 

The sixth driver relates to the reduction of 
achievement disparities among historically 
underserved groups. In the main, this standard 
would usually be focused on groups of 
particular racial/ethnicity characteristics. In 
the RSIs, there is usually only one racial/ 



ethnic group represented, i.e., Hispanic/ 
Latino, Black, White, Native American/ 
Indian, etc. Practically all groups attending 
these schools are historically underserved, 
poor and largely isolated from public learning 
institutions (museums, research universities, 
etc.), and children of parents who are poorly 
educated with few opportunities for 
postsecondary employment in situations 
requiring advanced science or math. 
Necessarily, one must really compare the RSI 
schools against other schools that are of a 
similar composition but that are not RSI 
participants or all other schools in the state. 
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Objectives and Research Questions of the Study 



The objectives of the study being reported in 
this document, The Rural Systemic Initiatives 
Evaluation Study, are listed below: 

1. to develop a system of indicators 
around each of the identified six 
drivers of educational system reform 

2. to determine the perceived relative 
importance and value of each of the 
drivers and indicators for reform in 
RSI schools in selected communities 

3. to determine the status of 
innovation/reform within selected 
communities with respect to factors 
thought to support or serve as barriers 
to innovation and education reform 

4. to determine the ways and the extent 
to which the perceived importance and 
value of the drivers and the 
characteristics of the community 
impact on systemic reform effort and 
student achievement in mathematics, 
science, and technology 

With the expansion of the program to include 
an additional consortia in 2000, the following 
research questions were added to the study. 



# What new or different forms of 
student assessment and teacher 
effectiveness have been developed and 
used as a result of curriculum 
transformation and an alignment with 
state or national standards in science 
and mathematics, and how were these 
developed and used? 

# What contextual factors (within and 
across collaboratives) serve to support 
reforms that result in or are associated 
with the development and 
implementation of standards-based 
curricula? 

# What processes and conditions are 
essential for effective partnerships 
within large-scale collaboratives that 
are characterized by large geographic 
distances between entities and 
multiple/potentially contrasting 
governance arrangements? 

# What considerations have been given 
to the use of technology for enhancing 
the accessibility and effectiveness of 
math and science instruction and 
communication within and among 
collaborative members? 



13 





Methodology of the Study 



As stated earlier, this study of the National 
Science Foundation’s Rural Systemic 
Initiative was intended to better understand 
these major efforts by consortia of school 
districts and other entities to reform/improve 
science, math, and technology education in 
rural America. Further, the study was 
designed to be conducted from the perspective 
of the rural school/community, not from the 
perspective of the administrators of the RSI 
consortia. The study was not intended to 
determine the effectiveness of individual RSI 
groups or even the success of NSF’s RSI 
program. Thus, at times, our observations and 
findings may appear counter to those who 
have taken a different slant on the study of Sis 
and/or other large-scale improvement efforts, 
including evaluators of individual projects 
who characteristically choose to identify 
extreme examples of observed success (or 
failure). We often refer to this as “cherry- 
picking,” which we think serves a purpose but 
may be not be the most solid basis for 
decision making with long-term potential for 
impact. 

We have already explained that while all the 
schools generally are from rural areas, they are 
both unique and inclusive. The uniqueness is 
generated from the requirement that they must 
be given an extreme “ruralness” index (8 or 9 
on the Beale Code developed and widely used 
by the U.S. Department of Agriculture), and 
they are unique in that a large percentage of 
the students must be designated as 
impoverished by U.S. Census definitions. 

To try to address a broad base of issues and 
research questions, we chose to implement an 
array of methods that are truly indicative of 



mixed-model methodology. In that regard, we 
used surveys of targeted groups on specific 
aspects of the study; case studies in 15 
communities; reanalysis of selected publicly 
available achievement test data; direct 
observations of classroom practices; 
interviews with a wide variety of stakeholders; 
focus group meetings with educators, parents, 
other and community personnel; and 
document reviews. Further, we couched the 
study in a loosely constructed application of 
the CIPP Model of evaluation with particular 
attention to the context and process elements 
of this model. 

The case studies were generally the most 
interesting and produced the most 
enlightening descriptions of the context of the 
community and the “gut feelings” of informed 
persons. The case studies were conducted 
over a period of 2-4 days by a team of three or 
more specially selected personnel. One 
member of each visitation team was a member 
of the Research Advisory Team. A manual of 
procedures, entitled the Site Visitors’ Guide 
for the Rural Systemic Initiatives Evaluation 
Study (1999), was developed as a part of the 
project. This manual/guidebook served an 
important function in maintaining focus and 
consistency in the case study visits. It 
included sample visitation schedules, 
background information on case 
studies/qualitative studies, and researcher 
worksheets for recording evidence on each of 
the six drivers. 

The Research Advisory Team 

The Research Advisory Team (RAT) was an 
important component of this study. This 
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group of ten professionals, all known for their 
specialized expertise and experience, were 
carefully selected and invited to participate 
prior to submission of the proposal to NSF. 
Every person on this team readily agreed to 
participate on first request. Members of the 
RAT are listed below along with their 
professional affiliation, state of residence, and 
area of expertise. 

Enochs . Larry - Oregon State University, 
(OR)-science education 

Hall. Gene - University of Nevada, Las Vegas, 
(NV)-science education and education reform 

Harris. Mary - University of North Dakota, 
(ND)-education reform and teacher education 

Jess. James - CAL Community Schools, 
(IA)-rural education and school 
administration 

Nachtigal. Paul - Annenberg Rural Challenge, 
(CO)-rural education and education reform 

Newlin . Joe (deceased) - National Rural 
Education Association, (CO)-rural education 

Oliver. Steve - University of Georgia, 
(GA)-science education 

Sanders. Jack - SERVE, (NC) - rural 
education and educational research 

Stufflebeam . Daniel - Western Michigan 
University (Ml)-evaluation 

Stephens. Robert - (formerly University of 
Maryland), (OK)-rural education and regional 
service centers (Succeeded J. Newlin) 



Webster. William - Dallas Independent School 
District, (TX)-testing and achievement 
assessment 

The RAT served in various capacities 
throughout the study, sometimes working as a 
group and other times as individuals or 
subgroups, depending on the needed area(s) of 
expertise. For example, as a group, they 
identified a long list of possible indicators for 
each of the six drivers and served as 
participating subjects in the Delphi technique 
used to identify those for which there was at 
least 80 percent agreement after two rounds of 
the Delphi procedure. (A full report of this 
effort is referenced as A Report on the 
Identification and Validation of Indicators of 
Six Drivers for Educational Systemic Reform 
by Russon and Horn, August 1999.) In other 
instances, several RAT members reviewed 
and helped assure validity of survey 
instruments, reviewed data and offered 
interpretations, planned additional substudies 
as questions and issues arose, provided 
feedback on publications and presentation 
materials, and served as members of the case 
study visit teams. In total, the Research 
Advisory Team, which was composed of some 
of the best minds in their particular fields, 
provided valuable assistance and added to the 
project’s validity and credibility. 

The Case Studies 

Small teams of professionals with 
complementary expertise and experience 
conducted case study visits to obtain a 
snapshot of the schools and communities and 
to understand how the systemic reform effort 
was being received and applied in this 
particular setting. The common goal was 
systemic reform and not the evaluation of one 
approach or another. 
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From a list of schools given to Dr. Horn by 
each consortia administrative head (PI or PD), 
two schools/communities were selected for 
each of the initial three RSI projects and three 
were selected from each of the three local 
projects included in the expansion of the 
evaluative study. The primary criteria for 
selection included (1) active participation in 
the project, (2) includes grades K-12, and (3) 
typical of rural schools in the projects. A list 
of the selected schools by consortia follows. 
(Schools chosen for case study sites were 
given the option of not having their name used 
in public documents and none requested such 
consideration.) 

! UCAN-Gila Indian Reservation (AZ) 

and Wagon Mound (NM) 

! Delta RSI- Humphreys County (MS) 

and East Feliciana Parish (LA) 

! Appalachian RSI-Cocke County (TN) 

and Rockcastle County (KY) 

Three study sites were selected for each of the 
latter three consortia. A more purposeful 
selection occurred here, because there was a 
desire to select schools/-communities with 
particular characteristics, which will be 
discussed later in the findings. The study sites 
for the Texas, Michigan, and Coastal RSIs are 
shown below. 

! Texas RSI-Clarendon (TX), Carrizo 

Springs (TX), and Pittsburg (TX) 

! Michigan RSI-Nah-Ta-Wahsh Public 

School Academy (MI), Baldwin (MI), 
and Whittemore-Prescott (MI) 



! Coastal RSI-Charles City County 
(VA), Marion (SC), and Elizabeth 
City (NC) 

After being selected as a potential case study 
site, Dr. Horn or his designate made an initial 
visit to each study site to discuss the purpose 
and plan for the case study, to address any 
concerns of the local school district, and to get 
official approval for the visit. Of the school 
districts nominated, only one did not grant 
approval for the visit; that one simply was 
nonresponsive to repeated efforts to meet with 
the superintendent. 

The case study teams were composed of one 
or two project staff members and a member of 
the RAT. With the Site Visitors’ Guide 
providing a blueprint, each case study was 
conducted over a period of 2-4 days. In the 
case of the Coastal and Michigan RSIs, two 
visits were conducted, since these RSIs were 
in their earliest stages of operation and we 
wanted to give additional time for 
development before completing the 
observations and finalizing the report. 

Whenever possible, the team members 
obtained lodging and had their meals in the 
local community in order to obtain some 
informal input and to get a sense of the 
context of the school and this community. 

A draft of each case study report was shared 
with the designated contact persons in the case 
study schools with instructions to read them 
for accuracy and to provide suggestions for 
improvement. In the main, feedback consisted 
of corrections of titles or names and/or other 
minor modifications. In no case were there 
objections to the observations, perceptions, or 
findings as expressed in the reports. 
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A final version of each case study was made 
into a separate report, with sections describing 
the history of the development and education 
of the area, socioeconomic information, 
description of the school/school district, 
involvement and major activities of the RSI, 
student achievement and accomplishments, 
and observed presence of the six drivers of 
systemic educational reform. Each report is 
available on the project’s Web site 
< http://www.wmich.edu/evalctr/rsi/ >. 

The Surveys 

Reports on two surveys are included in this 
report. The first was administered among 
stakeholders in schools and communities 
where case studies were conducted. This 
survey focused on perceptions regarding the 
drivers and other aspects with the realm of 
systemic reform. The full results of this survey 
are reported in two volumes ( A Summary of 
RSI School Personnel’s Perceptions of the 
Drivers for Educational Reform, April 2001 
and A Summary of RSI School Personnel’s 
Perceptions of the Drivers for Educational 
Systemic Reform-Part Two , September 2002). 
In total, there were 797 respondents to this 
survey, with 524 from ARSI, UCAN, and 
Delta sites and 272 from Michigan, Texas, 
and Coastal sites.) 

Another survey was administered to schools 
within five consortia that had been identified 
as “having participated with a reasonable level 
of involvement for the course of two or more 
years by the principal investigator (PI).” 
(UCAN was dissolved as an operating 
consortium at this time and was not included 
in this effort.) This loose definition was 
necessary because participation in an RSI 
consortium was voluntary after eligibility was 
determined. 



This survey was sent as a school district 
packet with instructions to distribute a copy of 
the survey to “a building principal”; the school 
district RSI contact”; and the “local facilitator, 
teacher partner, or local person who has 
substantial involvement in the RSI program.” 
A total of 195 school district packets were 
distributed, and the response rate from 
individual groups ranged from 11 percent 
from Coastal RSI “local facilitators” to 80 
percent for Michigan contacts. In summary, 
completed and usable survey forms were 
received from 79 “administrative contacts,” 45 
“school principals,” and 55 “local facilitators” 
for a total of 179 respondents This survey was 
intended to obtain perceptions and 
information about the RSIs’ impact on various 
aspects of the school program and a 
comparison between what they expected and 
what really happened. 

The Substudies 

Two major substudies of the project are 
reported in this document. Both related to 
comparisons of student achievement between 
a selected group of RSI schools and a 
comparable group of eligible but not 
participating RSI schools that were matched 
on selected variables (enrollment size, 
minority representation, and percentage of 
economically disadvantaged students, etc.). 
One substudy focused on Texas and the other 
on Kentucky. These two states were selected 
because it seemed that the RSI serving each 
state had demonstrated a substantial amount of 
growth and maturity at the time of the time of 
the substudy (-2004), a state testing program 
was fully implemented, and state test data and 
other school information were accessible 
through public interconnections. An 
additional and smaller study of Mississippi 
schools was made in an attempt to determine 
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the extent to which the initial findings were 
state specific. 

Various other data gathering techniques were 
used in the overall evaluation project, 
including review of documents generated by 
project personnel, external evaluators, internal 
assessment staff members, and others; 
classroom observations; focus group 
meetings; and interviews/discussions with RSI 
project and NSF program personnel. 
However, we are confining this report to those 



elements and procedures described above, 
which we believe more directly reflect these 
systemic reform efforts from the perspective 
of the local schools and communities. The 
case studies provided a wealth of information 
about the context and its importance in these 
poor, rural communities and the perceptions 
of the input and impact of the effort. These 
points are included along with additional 
information in the next section. 
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The Findings 



The findings from this extended study of the 
Rural Systemic Initiative of the National 
Science Foundation are based on a mosaic of 
data sources and investigations that span some 

19 states, each with its own set of standards 
for instruction and definitions of achievement. 
We used an array of mixed methods, which 
produced both quantitative and qualitative 
results, in an effort to understand the context 
of this uniquely defined set of schools and 
communities, to observe how each RSI and 
each school approached “systemic reform,” to 
assess the extent to which reform/ 
improvement has occurred in terms of the 
presence of the drivers, and to determine what 
factors support or serve as barriers to systemic 
reform. 

We began this effort without preconceived 
notions about the NSF program, and we did 
not restrict ourselves to considering a set of 
research hypotheses. We produced more than 

20 separate reports; generated hundreds of 
pages of field notes; reviewed many sets of 
test data; and collected stacks of documents 
and other materials describing individual 
RSIs, school districts, and individual schools. 
Whenever possible, we have verified 
observations and findings through 
triangulation of data sources and 
methodologies. As Sharan Merriam 
emphasized in a book published in 1992, we 
are not beginning the analysis of data, we are 
organizing and refining our findings and 
stating our discoveries. And, of course, we 
will always wonder about the extent to which 
we can generalize our individual findings and 
to what extent there are confounding factors 
that make that goal an impossibility. Thus, we 
will describe our findings with as much 



explanation of the context as reasonably 
possible. In the remainder of this section, we 
have organized our findings around some 
simply stated questions to help ensure both 
interest and understanding. 

Who did the RSIs serve? 

The regulations for participating in the Rural 
Systemic Initiative program of the National 
Science Foundation are specific and 
restrictive. Basically, to be eligible, a school 
must be classified as highly rural and poor, 
with at least 30 percent of its students 
identified as impoverished. However, this is 
a grant program and not an entitlement 
program, i.e., a school had to be located in an 
area for which a successful proposal had been 
submitted to NSF and had to be invited to 
participate. Aside from the Tribal Community 
Colleges, the initial areas were Appalachia, 
the Delta, the southwest (Utah, Colorado, 
Arizona, and New Mexico), and Alaska. We 
chose to focus on the Appalachia, UCAN, and 
the Delta, because we perceived these to be 
more closely affiliated with the schools 
serving the “traditional” rural communities of 
this country. For Appalachia and the Delta 
projects, this was an accurate perception, but 
it was only partially accurate for UCAN. In 
the UCAN project, a number of Tribal groups 
spanned across communities and had multiple 
governmental jurisdictions; thus, some of the 
sites were anomalies. 

The communities ranged across a broad array 
of peoples, with the majority of them 
economically poor and with little hope of 
major improvement in the near future. In 
some instances, only 1 person in a large 
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extended family had regular employment and, 
possibly by default or bad luck, was the sole 
support for 15 or more persons. On the other 
hand, almost every community had well-to-do 
individuals and families that enjoyed 
considerable wealth and property. Each 
community had its own unique history, which 
is richly described in the case study reports. 

A rather common finding was that there was 
a comparatively low level of educational 
attainment (less than a high school diploma) 
among many parents of children in the 
schools. As such, they were limited in their 
ability to seek higher paying employment or 
gain entry into technologically-based 
positions. In the main, they were destined for 
employment in low-paying service jobs, 
seasonal work in agriculture or food 
production enterprises, or other hourly work in 
industries such as piecework in local factories. 

Because of one RSI’s location near a major 
theme park for entertainment, employment for 
poor and not-well-educated persons was 
available in neighboring towns. In one 
instance, a county had a number of state 
institutions (prisons and a mental hospital), 
which provided a number of jobs for locals 
and a career goal for many students. 

It should be pointed out that some high school 
graduates sought entry and reportedly enjoyed 
some success in largely regional state 
universities and community colleges. 

There was greater racial/ethnic diversity 
across the consortium than there was within 
case study schools. For example, a consortia 
might have a mixture of schools, some largely 
Hispanic, some almost entirely white, and 
others with an almost totally African- 
American population. However, individual 



schools show little diversity. In Mississippi 
and in some other locations, the public 
schools serving RSI schools had an almost 
100 percent African-American enrollment, 
while nearby private schools or “academies” 
served the white members of the community. 

We commonly think of diversity in terms of 
race or ethnicity, but there are other forms of 
social and economic diversity. In many of the 
communities, there is clear evidence of the 
“haves and have-nots.” In the South, i.e., the 
areas served by the Delta RSI and portions of 
the Coastal and Appalachian RSIs, one finds 
families that have controlled the wealth of the 
area for hundreds of years as well as some of 
the “newer” wealth enjoyed by professionals 
who have moved to estate-type housing and 
others who have become successful in the few 
businesses or industries of the area. School- 
age children of the more wealthy families 
usually attend some form of private schooling. 
So, while the region has some diversity, the 
schools are considerably less diverse. 

In Appalachian and Texas RSIs, the schools 
more accurately reflect the composition of the 
community, since there are fewer private 
schools available and many of the 
communities are homogeneous in regard to 
race. Also, there is some evidence that there 
is less variance in family income. However, 
every community has families that are very 
poor and a few that are quite wealthy by local 
standards. 

Probably the most common type of school 
district was a “county school district” that was 
the latest stage of consolidations of smaller K- 
12 school districts or possibly K-8 school 
districts. Forced consolidations have 
historically resulted in bitter disputes among 
families and often within families, loss of 
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identity for one or more communities within 
the county, and a general loss of support for 
schools across all segments of society. Such 
is the case in some of the communities served 
by these RSIs. However, in one case the 
current school district is the result of an active 
process in which multiple K-8 districts wanted 
to have a common high school. In another 
instance, the community had been a summer 
retreat for well-to-do minorities from 
regionally located families, but with the retreat 
of this industry, the small summer homes were 
becoming “cheap” full-time residences for 
persons from nearby cities. 

The UCAN RSI contained a range of schools, 
including regular public schools, Board of 
Indian Affairs (BIA)-run and BIA-supported 
schools. Tribal schools, and semiprivate 
academies of one form or another. The Gila 
River Reservation in Arizona was one case 
study site. Along with multiple governmental 
jurisdictions and sources of financial support, 
it served only Native American children 
enrolled in grades K-8. There were no high 
schools on the reservation. Quality of housing 
and family income varied considerably, as 
there were some employment opportunities on 
the reservation that provided stable incomes, 
while many residents relied on some form of 
government support. A financially successful 
casino provided some jobs and a considerable 
amount of income for the tribe, which was 
investing this into the infrastructure of the 
community rather than dividing it up among 
registered members of the tribe. 

Another aspect of “rural” is geographic 
isolation, and this was clearly evident in our 
study. While we generally think of distance as 
the determiner of isolation, other factors 
produce this effect, i.e., severe winter 
conditions, mountain ranges, rivers, etc. 



However, isolation can occur by simply not 
feeling safe or secure in leaving your home 
environment, as explained about reservation 
children, hill people in Appalachia, Hispanics 
in small communities in the southwest, and 
African-American children in the Delta. 

Isolation also occurs when you have no 
physical means to travel to a larger 
city/community, which is not rare among poor 
residents. There are virtually no public 
transportation systems in rural America. At 
the same time, distance may not be a factor of 
isolation in a west Texas town where people 
may drive 50 to 75 miles for dinner at a 
restaurant or to attend a sporting event. While 
there may be a general feeling of isolation, 
this is not to say that people living in rural 
America are not aware of world news, thanks 
to satellite communications, cell phones, etc. 

The students in the RSI schools had relatively 
high attendance rates, except for those who 
might live in areas that have winter weather 
conditions that close roads for periods of time, 
i.e., certain mountain regions of Appalachia, 
the upper reaches of Michigan, etc. An 
exception to high attendance rates would be 
among high school students in many of the 
reservation schools and to some extent in the 
highly Hispanic communities of Texas and the 
southwest if you consider dropout as a form of 
nonattendance. Certainly, elements of the No 
Child Left Behind legislation and the 
associated regulations will likely have an 
impact on both actual attendance and 
definitions of dropout rates. 

In summary, the RSI serves students that it 
purports to serve in its eligibility 
requirements, i.e., poor children who live in 
rural areas. However, the program is not 
available to all rural children, and since it is 
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not an entitlement program, someone or some 
group must develop a consortium and write a 
program that is funded by NSF to be a 
participant in the program or receive services 
through the program. Beyond this, the 
communities are generally void of potential 
employment that would make math and 
science priorities and role models for student 
to emulate in their K-12 studies or career 
planning. Adults are relatively poorly 
educated with half or more in many instances 
without high school diplomas. Jobs of any 
type are limited, and those that might be 
available are low paying with little 
opportunity to advance. Some communities 
are without grocery stores, medical facilities, 
and other businesses or service providers that 
are taken for granted in larger communities. 
Yet, there is a sense of pride in rural 
communities that is reflected in parents and 
others in attendance at school events, 
statements made by all types of persons in the 
community, stories in local newspapers, etc. 

Since teachers are targeted groups in the RSI 
program, it might be well to mention them in 
this section. For some unexplained reason, 
most of the RSI activities that we observed 
were in elementary or middle schools. States 
generally have a K-8 type of certification that 
allows a teacher to teach all subjects within a 
self-contained classroom. Obviously, this 
situation is ideal for most elementary schools 
with students in grade six or below, but 
middle schools often provide instruction in 
specialized classrooms — i.e., math, science, 
social studies, etc. — and this may require a 
different type of certification. 

Shortages of certified and qualified math and 
science teachers are well known, and this is 
particularly true in rural areas. Thus, many of 
the teachers teaching science and math in 



middle and high school are provisionally 
certified. In effect, this means that they do not 
have the qualifications for that subject area, 
but they may be certified in another area. 

Provisional or emergency certificates are 
usually issued for a period of 2-3 years and 
often with a renewal option. For example, a 
teacher certified to teach social studies may 
apply for provisional certification to teach 
science with an expectation that he/she will be 
working toward full certification in that area 
over the next 2-3 years. Schools are granted 
approval to hire a provisionally certified 
teacher if they can document that a fully 
certified teacher is not available. 

At one of our case study sites, every math and 
science teacher was provisionally certified; 
and in conversations with individual teachers, 
there was not a clear indication that they were 
seriously working toward full certification. 
The annual turnover of teachers in this school 
was about 70 percent per year, so they would 
either be teaching elsewhere in 2-3 years or 
they would be assigned to another subject area 
with a different provisional certificate. A 
student could go through every science and 
math course in this school and never have a 
teacher who was certified or even minimally 
qualified to teach math or science. While 
school administrators can be criticized for 
their “creative” methods to “legally” place a 
teacher in each classroom through use of 
provisional (or unqualified) teachers, what 
choice do they have? Teachers are simply not 
available to teach science and math in many 
rural communities. 

When we added the latter three RSIs to our 
study, we decided to seek out case study sites 
that might be typical of expected groups of 
schools within the consortia and ones that 
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have recognizable characteristics, as discussed 
in rural education circles and with the advice 
of selected members of the Research Advisory 
Team. For example, in the Texas RSI we 
sought sites that were typical of west Texas 
cattle, oil, and gas country; one that was 
relatively near the Mexican border with a high 
percentage of Hispanic students; and one that 
was heavily influenced by a single large 
industry that attracted large numbers of 
“outsiders” as employees. For the first 
category, we identified Clarendon, a 
community near Amarillo that has a local 
community college and serves a largely ranch- 
based economy, but a number of the residents 
commute to jobs in Amarillo. 

Carrizo Springs was the site selected to 
represent many communities in the southwest 
that have a mixture of long-time families of 
Mexican/Spanish heritage as well as a number 
of more recently arriving residents from 
Mexico. In this particular case, the school 
district had recently undergone a consolidation 
process. 

For the single or one predominant industry 
community, we selected Pittsburg, located in 
eastern Texas relatively near the Arkansas 
border. This community is the home of a 
huge poultry growing and processing industry, 
founded and primarily owned by a long-time 
member of the community. This industry has 
attracted low-paid workers from other areas, 
including Mexico, who brought a different 
culture and set of education experiences and 
expectations to this rather traditional small 
rural community. 

This information is provided to the reader as 
an indication that there was no attempt to 
study every community or even to find an 
“average” community, which simply does not 



exist in these geographically large and diverse 
settings of people and cultures. We thought it 
was far better to identify typical or 
representative communities of those that we 
might find in each consortium. Further, we 
wanted to conduct case studies in schools and 
communities that had been active participants 
in an RSI. Membership was voluntary, as was 
participation; and we thought it to be far more 
valuable and appropriate to devote our efforts 
to those sites that were or had been heavily 
involved, which was verified by the PI/PD of 
each project. 

In summary, the RSIs that we studied served 
a widely diverse set of schools and 
communities. Some RSIs were fairly 
homogeneous in terms of types of schools, 
racial/ethnic diversity, and socioeconomic 
status, while others, such as Texas and 
Michigan, included a variety of schools that 
were typical of those in their area. 

What was the nature and form of the RSI 
assistance/programmingl 

Each RSI worked with schools and other 
partners in the development stage to identify 
needs and programming that were intended to 
meet N SF’ s goals for the RSI program. About 
three different approaches appear to prevail, 
one most closely associated with each of these 
RSIs — Delta, Appalachian, and Texas. To 
help understand these approaches, the goals of 
each, as extracted from their literature, are 
listed below. 

The goals for the Delta RSI are these: 

# Enhance the K-12 learning 
environment in the targeted counties 
and parishes to address each child’s 
needs and promote each child’s 
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achievement in science, mathematics, 
and technology. 

# Increase the capacity of local 
communities to build and maintain 
quality science, mathematics, and 
technology educational programs for 
each community’s children. 

# Establish mechanisms to champion 
policy development and implemen- 
tation at local and state levels in 
support of sustained reform in science, 
mathematics, and technology. 

# Create regional infrastructure that will 
utilize intra-and interstate alliances to 
develop sustainable regional improve- 
ments in science, mathematics, and 
technology. 

Three strategic goals for the Appalachian RSI 

are listed: 

# Strengthen the knowledge and skills of 
teachers in grades K-12 so they can 
teach mathematics and science more 
effectively. 

# Establish a timely and coordinated 
system for helping schools enhance 
their capacity to deliver active, 
standards-based teaching and learning 
environments on a long-term basis. 

# Build regional partnerships, local 
leadership, and local community 
involvement and support for long-term 
educational improvement. 

The Texas RSI goals, stated as “attributes,” 

mirror the six NSF drivers of systemic reform: 



# Successful implementation of the 
Texas Essential Knowledge and Skills 
(TEKS), the state standards-based 
curriculum framework, in mathematics 
and science. 

# District policies supporting TEKS 
mathematics and science implemen- 
tation and systemic reform through 
campus administrative support and 
alignment of district improvement 
plans with TRSI attributes. 

# Alignment of resources to support 
systemic reform efforts and TEKS 
mathematics and science implemen- 
tation. 

# Stakeholders’ commitments to 
systemic reform of district mathe- 
matics and science education. 

# All district students reaching high 
academic standards. 

As explained earlier in this document, this 
study was conducted in a manner that focused 
on the RSI from the perspective of the local 
school and community. Thus, we will not 
attempt to define or describe the consortial 
efforts from the projects’ administrative 
perspective or stated intentions. 

From what we heard at the local level, not 
much money ($10K -15K) actually filtered 
down to the school district level. Thus, 
whatever assistance and support derived from 
the RSI program are perceived to be mainly in 
terms of 

# leadership capacity building for 
education reform 
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# raising the awareness of the 
importance of science and math 

# assistance in aligning curricula with 
state standards and benchmarks 
reflective of state-mandated tests 

# teacher enhancement through group 
workshops and directed professional 
development 

However, each RSI consortium developed 
somewhat unique approaches to provide these 
services. While these will be discussed more 
fully in later sections of this report, the single 
characteristic that seemed to define each of 
these three RSIs is as follows: 

# Texas — assistance to individual 
teachers in the form of class quantity 
materials and focus on state-mandated 
tests 

# Appalachian — resource teacher and 
curriculum review process 

# Delta — services provided by teacher 
leaders and field coordinator provided 
services 

The Coastal RSI showed a good deal of 
similarity with the Appalachian RSI, but on a 
considerably smaller scale. The Michigan RSI 
served rather distinct types of schools, each 
with more clearly defined differences in 
needs. The approach was to identify a 
consultant who would work with a school on 
a long-term basis, with a focus on meeting 
these locally identified needs. [A reverse study 
visit by NSF staff and representatives forced 
this project to change its approach and 
leadership personnel or lose its funding. This 
caused a great deal of disruption within the 



consortium and added to the participating 
schools’ mistrust of the long-term 
commitment made to them in the initial 
planning development stage.] 

The UCAN project basically dissolved as a 
functional entity within the first few years of 
its operation. During its time of existence, it 
provided some much-needed assistance in 
curriculum alignment and a communications 
network across several states and stakeholder 
groups. 

Although an attractive set of materials related 
to community involvement in the reform 
effort was developed by ARSI, there was little 
evidence of consortia-wide efforts that were 
viewed as effective by local schools. 

The RSIs operated as umbrella organizations 
with relatively small staffs. In many instances 
the names of the Pis or PDs of the RSIs were 
not known in the local schools, since they 
were most familiar with the individual(s) who 
conducted workshops or provided on-site 
consultation services. For example, in the 
Michigan RSI, a math consultant was assigned 
to one school and worked intently with the 
math teachers. He was known by the front 
office and freely moved around the building 
from room to room during the times he was at 
the school. In the Texas RSI, specialized 
consultants made regular rounds to some 
schools and even responded to individual 
teachers’ requests for laboratory-type 
materials to assist in teaching a lesson. These 
local consultants or persons who provided the 
assistance were in most cases employed or 
contracted by the RSI but did their work in 
local schools, either with some direction from 
the PD or completely based on an agreement 
with the local school. 
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How did local schools view education 
reform? 

There are really two major issues to discuss 
here. One is what local schools and rural 
communities think of “outsiders” who have 
come to “reform” them, and the other issue is 
of what value or importance these same 
entities view the six drivers touted by NSF as 
evidence or indicators of reform in action. 

Rural literature would lead you to believe that 
rural communities are skeptical of outside 
influence and would resist efforts to take 
control of their schools away from them. To 
the first issue, the leadership of the RSIs 
displayed an awareness of the rural culture 
and the prerequisite need that participation 
was voluntary and could be terminated at any 
time. Plus, not much was offered by the RSI 
and not much was expected. When 
consultants were used, which was often, the 
RSI attempted to select individuals known by 
the local community or to identify persons 
with reputations that supported their work 
with local rural schools. In essence, the myth 
that rural communities are resistant to outside 
influence is only partially true, if at all, in 
regard to the RSIs and their work in these 
consortia. 

The other point about whether the six drivers 
are recognized as indicators of reform and to 
what extent they are valued in local rural 
communities is addressed by the data found in 
two substudies conducted by WMU. The full 
results of these substudies can be found in A 
Summary of RSI School Personnel’s 
Perceptions of the Drivers for Educational 
Systemic Reform (April 2001) and A 
Summary of RSI School Personnel’s 
Perceptions of the Drivers for Educational 
Systemic Reform-Part Two (September 2002). 



On a five-point Likert-scale response sheet (5 
= Strong agreement and 1 = Strong 

disagreement), no statement related to the 
validity of a statement about the drivers or 
elements thereof was marked lower than 3.69 
by the initial RSIs of this study and 3.23 
among the latter three consortia. Ironically, 
the one rated lowest by both groups related to 
“broad-based support from parents, 
policymakers, institutions of higher education, 
business and industry, foundations, and other 
segments of the community for the goals and 
collective value of the program.” The drivers 
related to a comprehensive standards-based 
curricula, coherent and supportive policies, 
and administrative support were the highest 
valued drivers. 

In terms of the locus of control for the drivers 
or activities/responsibility for accomplishing 
what the driver reflected, this was fairly 
consistently placed with the teacher with the 
exception of policies and broad-based support 
for student achievement. Noticeably absent 
from being listed as having substantial 
responsibilities were building level 
administrators (principals), which appears to 
be inconsistent with most effective schools 
research. However, we noticed that the 
building principal was not a major player in 
most RSI activities, especially when a 
“teacher partner” or someone with 
responsibilities for leading the local effort and 
providing on-site assistance for teachers was 
a part of the overall scheme. It is not clear 
whether this was by design or whether it was 
an unintended outcome. In on-site interviews, 
principals seemed to be minimally informed 
or involved in RSI work. 

In summary, we found no evidence that there 
is an inherent resistance to outside assistance 
for this program among rural schools or 
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within rural communities. It is also fairly 
evident that the community (parents and 
others) was not very aware of the RSI as a 
major intervention in the local schools. This 
may be the result of little self- promotion by 
RSI projects, the reality of the RSI as being 
one of several ongoing school improvement 
efforts, or the simple fact that these rural 
communities do not have much involvement 
or interest in the schools at this level, i.e., 
above the experiences of their children. 

What elements support large-scale 
collaboratives? 

The National Science Foundation has 
developed and supported a number of 
initiatives for school improvement over a 
period of more than 40 years. Some have 
focused on the improvement of teachers 
through workshops and summer and academic 
year institutes. Other efforts centered on the 
development and implementation of 
inquiry/laboratory-oriented curricula for 
grades K-12 and for other specialized groups. 
Additional programs involved the 
enhancement of teachers’ knowledge in 
subject matter fields as well as improved 
instructional techniques. Probably none have 
been designed to make such sweeping changes 
as those in the area of systemic reform — state, 
urban, and rural. 

The rural component differs from the other 
two components in significant ways. The 
state systemics were operated by or in very 
close cooperation with an existing agency 
(state departments of education) that had 
resources and legal supervisory 
responsibilities over the schools it was to 
serve. The urban systemics dealt with one 
legal organizational unit (the urban school 
district), which also had a number of controls 



and oversight responsibilities. In contrast, the 
RSIs served vast geographic areas made up of 
many jurisdictional agencies/school districts 
and often multiple states. 

Most, while not all, of the RSIs were founded 
by university-related personnel without formal 
ties to local schools, rural communities, or the 
state departments of education that had 
oversight responsibilities. The developmental 
grants were supposed to provide resources for 
building these collaboratives, which required 
the entrepreneurship and effective 
organizational skills of one or a few 
individuals and the willingness of many others 
to engage in planning a proposal for a 
collaborative that was new and untested. 

From the beginning, the RSI 
collaboratives/consortia struggled to find a 
niche in the overall and ongoing school 
improvement efforts. Coincidentally, many 
states were engaging in developing standards, 
mandated student testing programs, and 
accountability systems for local schools. The 
RSIs recognized the opportunity and quickly 
became a project to help local schools align 
math and science as well as the other curricula 
with state standards. In other words, they 
identified a common need across schools and 
offered expertise and the capacity for 
providing leadership in this area. Schools 
welcomed this assistance, and by chance or 
good timing, the RSIs found a home. 

As stated earlier, the RSIs received a 
substantial amount of money from NSF, but 
with the size of the coalitions and the number 
of participating school districts, there was 
little financial support that could or would go 
to individual schools at the local level. Each 
RSI developed a set of services it would 
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provide, and as long as the teachers who were 
the primary targets were satisfied, there was 
little reason to question participation. 

Because of the size and complexity of the 
RSIs, there was an almost instant move to 
decentralize services and responsibilities. 
Thus, local schools identified a consultant or 
a resource member of the team as “THE RSI.” 
School administrators were involved only in 
secondary ways or almost not at all in some 
cases. Inquiries about RSI activities in many 
of the schools resulted in the question being 
referred to the “teacher partner” or the 
designated contact in the school or school 
district. Interestingly, we found the school 
contact to be in the form of various persons, 
i.e., a superintendent, the high school 
counselor, the curriculum director, a teacher, 
the “grants” or federal projects coordinator, 
etc. 

Coordinating councils involving project staff, 
school representatives, and others from 
various public agencies and the private sector 
were established by most RSIs, but when 
asked about specific operational and 
programmatic decisions, they did not appear 
to be critical to the process and certainly not a 
major decision-making body. In the main, 
they served as advisory bodies and a vehicle 
for communicating information about the 
consortium to their representative groups. 

The PI or PD of the project was a key to the 
success of the RSIs. In some cases, the role 
was that of a CEO; in others, the PI was 
involved in day-to-day activities. Essentially, 
either a PI or a PD needed to be actively 
involved in the project and able to make 
decisions without taking it to a higher level. 
Some projects had very active Pis who liked 
to be out front with the project and to interact 



at a school level or at the classroom level if 
the opportunity presented itself. Someone had 
to be the motivator, cheerleader, and a good 
communicator among the various partners and 
the funding source. 

Another critical element of a successful RSI is 
the ability to recognize needs and respond 
appropriately, whether that is to provide some 
information, offer a workshop on a specific 
topic, select a person who will be an effective 
consultant to a local school, etc. 
Schools — particularly rural schools — have had 
somewhat bitter experiences when they have 
been promised something and then nothing is 
ever heard from that group again or they are 
nudged out of the picture by larger and more 
powerful groups. 

We expected to find sophisticated or at least 
innovative use of technology within the RSIs 
for administrative purposes as well as for the 
delivery of services, but that was not the case. 
In fact, we cannot cite an instance in which an 
RSI could be given credit for the invention of 
a creative use of technology in its operations. 
While one RSI tried to develop technology as 
a centerpiece of its operation, this was 
abandoned within a short period of time. 

As a final word about what it takes to operate 
a large scale collaborative, it might be 
condensed to a rather simple statement — 
identify needs, provide timely and reasonable 
responses, listen to the partners, provide 
skilled and knowledgeable people who are 
able to work with teachers, let local schools 
set their course, and have the collaborative 
administered by a person who is creative, 
flexible, and willing to share successes with 
those who are most responsible for them 
occurring. 
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Impact 



The impact of the RSI program and its 
scattered projects or consortia can be viewed 
from several perspectives. Individual project 
reports submitted to NSF and some external 
evaluators’ reports cite instances of 
considerable impact and improvement. Often 
these citations are supported by examples of 
extraordinary increases in achievement by a 
school within a specific time frame, 
exemplary activities by persons who have 
been motivated by their involvement in RSI 
work, testimonials by students, teachers, 
administrators, or parents, and a variety of 
other positive outcomes. However, we have 
tried to avoid cherry-picking isolated instances 
of impact and focused on evidence of impact 
that is found in our investigations through 
various methodologies, i.e., case studies, on- 
site observations and interviews, and test data 
reanalyses. Further, we are particularly 
conscious of our overriding attempt to study 
the RSI program from the perspective of the 
local community. Importantly, we primarily 
have focused our evaluation of impact around 
the six drivers established by NSF as the areas 
of evidence that school reform is occurring. 
At the same time, we must understand that 
schools have a number of ongoing efforts for 
improvement. In not all of these cases are 
these efforts coordinated or even have the 
same goals; therefore, we must consider 
participation in an RSI as one such 
intervention, and impact compatible with the 
goals of RSI does not necessarily prove that 
there is a cause and effect relationship. 

With regard to the implementation of a 
comprehensive, standards-based curriculum as 
represented in instructional practices, student 
assessment, and in every classroom. 



laboratory, and learning experience, it is not 
possible to say that this has fully occurred. 
However, it is possible to say that the RSI has 
played a major role in providing a base and a 
source of expertise for school improvement. 
As the RSIs were being developed and 
implemented, most states were requiring local 
schools to align their curricula with 
identifiable standards and benchmarks; and 
this would be carried forward into the state 
student assessment program. 

Federal monies were made available through 
state departments of education to accomplish 
this goal and, if not by statute, certainly by 
statements of consequences for 
noncompliance, the seriousness attached to 
aligning the curriculum with established 
standards was clearly understood. At about 
the same time, the RSIs came onto the scene 
with some expertise and a focus for aligning 
the science and math curricula. Thus, a 
common goal was established, and the result 
is an almost unanimous response. However, 
when you take the other parts of this driver 
into consideration, there is considerably less 
positive evidence. In schools where case 
studies were conducted, we found the entire 
range represented with regard to a standards- 
based curriculum being reflected in 
instruction. More likely, we saw a strong 
tendency toward teaching to the test of the 
specific state. Some teachers reported that 
they planned instruction according to the 
standards, while others were trying to “guess” 
the questions that would be on the state test 
and develop instruction around them. Others 
conducted periodic checks as to the extent to 
which the standards had been addressed 
within recent times. 
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Local schools that had received the services of 
curriculum reviews — conducted either by RSI 
staff members or other external consultants 
paid with RSI funds — perceive this to be a 
valuable effort and one that enabled them to 
improve both the breadth and depth of their 
curriculum. This practice was confined to two 
of the six consortia studied. In others some 
type of curriculum review did occur, and these 
also were deemed as valuable. 

As reported in one substudy (Stark, 2002), 
some of the changes that occurred in 
instruction include greater use of 
manipulatives; increased focus on concepts, 
problem-solving skills, and higher order 
thinking skills; more grouping and team work; 
an increase in the integration of the curriculum 
across all subjects; and an increased focus on 
students explaining what they were learning. 

Almost anything that included materials was 
called inquiry teaching, which is a far cry from 
the traditional definition of inquiry teaching. 
However, we did see teachers engaged in 
some creative questioning techniques and 
other methods that they said were learned in 
RSI workshops. 

There was some evidence that student 
assessment practices have been influenced by 
RSI activities. For example, teachers reported 
greater use of rubrics, open-ended response 
questions, checklists, etc., as a result of their 
introduction in RSI workshops (see p. 4, 
Stark, June 2002). In the main, we saw 
multiple forms of assessment in use. 

The driver dealing with the development of a 
coherent set of policies took some thought and 
understanding on the part of the research 
team. Official policies in most small schools 
or for that matter fairly large schools are 



customized adaptations of a standard set of 
policies often developed by a contractor, e.g., 
a state association of school boards. These 
policies are relatively general and do not deal 
with the specific types of policies that would 
be reasonably influenced by RSI involvement. 
However, we decided that a “policy” does not 
necessarily have to be written and included in 
a policy manual. Thus, we began to recognize 
that defined practices, procedures, or 
requirements that were disseminated, 
understood, and followed were in effect the 
same as a policy. 

In the site visit schools, about three-fourths of 
the schools were judged to be making 
moderate or greater progress toward having 
policies in place that support high quality 
math and science for every student. Such 
policies might relate to the number of credits 
for graduation, the types of courses that are 
required, regulations for teacher qualifications 
or requirements for completing professional 
development, use of resources/facilities, etc. 
Other instances directly reflective of a 
documented practice that could be covered by 
a policy were the regular review of test data to 
identify strengths and weaknesses in the 
instructional program and review of subject 
matter within a grade level or among teachers 
of the same course and vertically though the 
curriculum. 

The perceptions of the various groups 
regarding the impact on policies are found in 
a substudy report (Horn & Tressler, 2003) in 
which the mean response hovered around 3 or 
the midpoint of a 5-point scale. 

We saw little evidence that policies had been 
developed that pertained to an increase in the 
requirements/educational attainment for 
teachers or that modified the way teachers are 
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evaluated. In some cases, policies moved 
courses down to lower grades, i.e., biology 
from 10th grade for all students to 9th grade 
for advanced students, etc.; and these were 
said to be the result of comprehensive 
curriculum reviews. 

The driver pertaining to the convergence and 
usage of resources to support science and 
mathematics education was determined to be 
strongly evident in about one-third of the case 
study sites with the other two-thirds showing 
some visible signs of that happening. The 
problem is that schools have very few 
resources to reallocate within an operating 
budget and grant funds are mostly in the form 
of entitlement monies, which have “strings 
attached” as to how they can be spent. The 
most available dollars for choice assignment 
are probably within the category of 
professional development. Questioning about 
the availability of dollars to support requests 
for attendance in professional development 
activities produced rather a generally positive 
response. Plus, the RSIs could and did 
provide resources to support professional 
development. That said, the total amount of 
school monies for professional development 
was enhanced by the use of RSI monies. 

Other evidence of RSI activities influencing 
the convergence and use of resources could 
be found in the use of library monies and 
equipment monies that were often earmarked 
for science and mathematics. In summary, the 
RSI participation heightened the awareness of 
science and math and focused attention on the 
needs for science and math. Where science 
laboratories were found, they were lightly 
used. In one case, materials that had been 
delivered months, if not years, before were 
unopened and unknown to the science and 
math teacher of this school. Their discovery 



led to an internal debate about who should 
have priority on their use. In other cases, we 
saw some creative use of school grounds and 
a greenhouse for students’ projects. Student 
gardens and study plots for plants were 
observed at schools on the Gila River 
Reservation, which also served as a source of 
vegetables for families in an area with one of 
the highest rates of diabetes in the country. 

In one school, conversations with the library- 
media specialist revealed that she used any 
excess monies only for science materials. In 
another school, the school improvement 
committee could decide on where and how 
operational monies would be spent. During 
this period of time, science and math areas 
were given highest priority. 

The Texas RSI provided classroom-quantity 
materials for individual teachers to use. This 
is a very popular service and one that 
distinguishes this RSI as a teacher-focused 
venture. A PI of another RSI emphasized the 
point that its major goal was to “develop local 
leadership for science and math,” which 
requires a different priority for using project 
resources. 

On a scale of 1 (Strongly Disagree) to 5 
(Strongly Agree), stakeholders in the Coastal, 
Michigan, and Texas RSIs perceived that this 
driver (convergence of resources) was rated 
from 3.27 to 4.22 as an indicator that reform is 
occurring. In comparison with other drivers 
or elements of them, this was one of the 
lowest rated items. 

One particularly noteworthy activity 
conducted by an RSI was designed to teach 
school teams what certain grant areas 
permitted and how they could leverage their 
resources through careful use of multiple 



31 




grants for a common effort. For example, 
professional development funds were 
permitted by one grant area and equipment 
could be purchased through another one. 
Combined, they could buy equipment and 
conduct professional development activities to 
teach how they could best be used in the 
classroom. 

In summary, teachers normally do not have 
the authority to direct monies or the authority 
to determine how they can be used. However, 
the RSI initiatives seem to be opening their 
eyes to how they can let their needs be better 
known and how to support administrators who 
have to make decisions that direct scarce 
resources to areas of priority. 

Parental support and involvement in the 
schools and the schooling of their children has 
been a long-desired goal of education. Yet, 
seemingly there has been little widespread 
success in this area. Unfortunately, the RSI 
program has not found the “silver bullet.” 
Instances of local activities were successful in 
attracting parents to math or science nights or 
to science fairs, but sustainable programs that 
will greatly increase parental involvement 
were simply not found. ARSI’s guidebook 
appeared to have potential, but personal 
communications with leadership personnel of 
that collaborative confirmed our suspicion that 
little progress was being enjoyed. 

While we think the RSI effort was not very 
effective in this area, we would agree that 
“broad-based support and involvement” of 
parents and others would be evidence of true 
change and reform. This might be a 
particularly big challenge among the parents 
served by this program, because they generally 
have a low level of academic achievement 



(half or more without high school diplomas) 
and may be reluctant to become involved. 

The accumulation and use of a broad and deep 
array of evidence that the program is 
enhancing student achievement is the focus of 
another driver. The impact of RSI work in 
this area seems to be coupled with the reliance 
on achievement test data as an indicator of 
successful schools at the state level. Some 
states included in the six RSIs of this study 
have been using statewide and high stakes 
testing for years. Clearly, the consequences of 
poor results over a period of years are well 
known and understood. Thus, schools in 
those states generally spend a good deal of 
time studying the data, identifying problems, 
and developing strategies to improve. Some 
would argue that this is simply another case of 
the tests driving the curriculum and, as a 
result, the children’s experiences in the 
sciences are narrow and lack the opportunity 
for them to explore areas of interest and 
engage in inquiries that may require 
substantial amounts of time. Tennessee, for 
example, has embraced the idea of value- 
added assessment, and schools in that state 
receive data that help them understand school 
and teacher effectiveness. Other states 
provide local school data to explain how 
students do in specific areas of the tests, and 
this seems to be particularly useful in 
developing math skills in states like Michigan. 
Some of the professional development 
activities conducted by RSIs are focused on 
using assessment data for planning instruction 
and evaluating program effectiveness. Again, 
this has been a parallel result, since teachers 
are doing this in areas in addition to math and 
science; so the RSIs cannot be identified as 
THE reason, but they have added to the skill 
level of teachers and administrators in this 
area. 
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Overall, we see teachers and schools using 
multiple types of assessment. To some extent 
this reflects a broader array of evidence, but 
we do not see organized evaluations of 
instructional programs at the local or state 
levels. Informal evidence of success, such as 
student success at the next level, enrollment in 
courses that lead to career goals, etc., are not 
generally a part of decisions about the 
curriculum. 

The No Child Left Behind legislation has 
required states to disaggregate data by various 
groups, which provides a good basis for 
determining that all students are achieving. As 
written, it appears that we are supposed to 
know who the “historically underserved” are, 
and that is not the case in the schools and 
communities served by the RSIs. In reading 
other research in education, one could 
conclude that the historically underserved are 
minority students, but most of the schools 
served by these RSIs are relatively 
homogeneous in terms of race/ethnicity. In 
effect, the RSIs are defined to address the 
historically underserved, i.e., the poor, 
children of poorly educated parents, and 
others with racial and ethnic backgrounds 
substantially different from the majority in the 
classrooms. So, this driver seems to be a 
somewhat moot issue in NSF’s Rural 
Systemic Initiative, and it does not seem 
appropriate to consider the impact of the RSIs 
on this area. 

Overall, we think that the RSIs have had 
substantial impact on 

# alignment of the curriculum with 
recognized standards 

# raising the level of awareness of the 
need and the priority of science and 



math in the schools, particularly at the 
elementary and middle school levels 

# creating better understanding and use 
of multiple achievement data for 
curriculum planning 

# developing local expertise to serve as 
a resource for other teachers 

Certainly, one finds impact in other ways in 
specific locations. Individual schools and 
school districts accepted various roles in the 
RSI movement. Some were involved in the 
planning of the consortia and participated in 
developing the proposals to NSF. 
Reasonably, this group had developed a 
particular interest and ownership in the 
process. Others were invited to participate 
after overall funding from NSF was achieved, 
and this would put them into a different 
category as to ownership and interest. In 
some cases, one or two local teachers jumped 
at the opportunity to get involved, and they 
served as the nucleus of a local improvement 
body. There are instances in which a 
heretofore rather unknown teacher suddenly 
emerged as a leader among peers in the effort, 
and this is why it is so difficult and maybe 
inappropriate to judge a program across its 
total expanse. The stories of success (and 
failures) are at the local level. 

Impact on achievement is one of the most 
important considerations. Unfortunately, we 
have been unable to identify data that would 
indicate that there is a consistent and 
generalized positive impact on student 
achievement. While each state is different, 
there is generally a state-approved set of tests 
that are administered on a prearranged 
schedule. For example, using criterion- 
referenced assessment instruments, a state 
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might test grades 4, 6, 8, and 10 in math every 
year and test in science in grades 6 and 10 and 
at the end of the biology course. Further, 
some states and/or school districts administer 
an off-year or annual, nationally-known, 
norm-referenced achievement test. It is 
presumed that the criterion-referenced tests 
are based on what the children are “supposed” 
to learn, based on state-approved standards. 
Scores are most often reported as percentage 
of students who pass the tests at specified 
levels, such as novice, proficient, etc., while 
the norm-referenced tests provide reports on 
the percentile of students within a national 
pool and possibly the school or school district 
as a whole. In theory, this pattern of testing 
provides a means for accountability for state 
expectations and a reference for how well 
students within a state are doing in terms of 
achievement compared with students across 
the country (as defined by the norm pool for 
the tests). 

From on-site interviews with teachers, 
administrators, and others within the case 
study communities, there is a general feeling 
that students are doing better in science and 
math after (or as a result of) the RSI 
intervention. Queries for the bases of these 
feelings often focused on the increase in test 
scores for a particular grade level in a 
specified year, descriptions of individual 
student accomplishments, and “gut feelings” 
about student accomplishments. Seldom did 
school personnel or parent/community groups 
cite data from locally prepared or state 
originated school reports in support of their 
feelings or thinking. Specific questioning of 
school personnel revealed an array of 
interpretations of test data. More often than 
not, they were not very familiar with the data 
or how they should be interpreted. 



However, it was important to conduct some 
substudies to determine if RSI involvement 
had an impact on student achievement. For 
the substudies completed in 2004 after the 
Appalachian and Texas RSIs had been 
operational for several years, we compared 
student test data of RSI schools to similar RSI 
eligible but nonparticipating schools and all 
schools in the state. We examined and 
reanalyzed publicly available test data and 
compared groups and looked for trends. (Full 
reports of these studies are found on the 
project’s Web site at > http://www.wmich.edu/ 
evalctr/rsi/< . From the Kentucky substudy, it 
was concluded that the ARSI program may 
have had some impact on students’ 
mathematics and science achievement as 
measured by the academic index on the KCCT 
mathematics and science batteries. At all 
levels assessed (elementary, middle, and high 
school), the academic indices have steadily 
increased each year, although it is interesting 
that the matching school districts displayed 
similar trends of progress. Because both the 
ARSI and matching school districts 
experienced comparable positive increases, 
other factors besides the ARSI program may 
have contributed to increases. Disaggregating 
the 2003 mathematics and science academic 
indices for students qualifying for free or 
reduced price lunch revealed that ARSI 
elementary, middle, and high schools had 
slightly higher academic indices than both the 
matching schools and the state. 

From a Texas study conducted in a similar 
manner, it was concluded that the TRSI 
intervention made some impact in student 
academic performance, especially in 
mathematics after one and two years of 
programming. However, the Texas 
assessment system had been substantially 
modified, and it may take two or more years 
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of statewide testing to fully understand the 
extent of the impact. At this point, it appears 
that the impact may be rather minimal in some 
cases and not consistent over time. For 
example, the TRSI group led the other three 
comparison groups (non-TRSI, Regional 
Educational Service Center schools, and the 
state), however slightly, on the middle level 
science scores in 2001, while in 2002, the last 
year of the administration of this test, they 
were the lowest of the four groups. 

A similar study was done among a matched 
sample of 30 Delta RSI schools and 30 non- 
RSI schools in Mississippi. Each RSI school 
was manually matched on the selected 
variables that have historically been of 
concern in studies of this type. More 
specifically, these schools had an average K- 
12 enrollment of around 2,000 students, 80 
percent minority (African American), 80 
percent eligible for free or reduced price 
lunch, a special education population of 13 
percent, an expenditure of $6,000/student, and 
an assessed property valuation of 
~$25, 000/student; and 36 percent of the 
teachers had advanced degrees with around 8 
percent holding emergency certificates. 
Statistically, the 2 samples were determined to 
be equivalent. 

From data derived from files of the 
Mississippi Department of Education, a 
variety of 2002 test scores were extracted and 
reanalyzed for this substudy, including the 
Mississippi Curriculum Test (math-grades 6 
and 8), the CTBS (math-grade 6), and subject 
area tests (Algebra I and biology). Again, no 
statistically significant differences were 
determined. With other inquiries into these 
test data and others from previous years, there 
were no discernable indications that 
participation in the Delta RSI had affected the 
achievement of these students, as measured by 



these assessments. It might be important to 
note that even though these were selected 
schools, there were some variations among 
them. For example, the ranges on selection 
variables were 616 to 6,415 for enrollment; 33 
to 100 percent for African-American minority 
enrollment; 35.4 to 95.4 percent of students 
eligible for free or reduced price lunch; $3,317 
to 50,143 for valuation per student; and 2 to 
24 percent for teachers holding emergency 
certificates. 

Since the two groups were determined to be 
equal, we calculated a number of correlations 
between the selection variables and the 
various measures of science and math 
achievement for the total group (60 schools) 
as well as for the two subgroups (30 each). 
Consistently, there were large, negative 
statistically significant correlations between 
percentage of minority (African American) 
students and test scores and percentage of 
students eligible for free and reduced lunch 
and test scores. 

In summary, it is difficult to find clear 
evidence that there was an overall consistently 
positive (or negative) impact on student 
achievement as a result of schools/school 
districts’ participation in an RSI. Because of 
a number of other ongoing programs that are 
school specific, it is even more difficult to 
attribute changes to the RSI movement alone. 
Even as cautious as the above summary 
statements are, they still could be challenged. 
Again, we emphasize that individuals or 
specific schools might have been impacted 
substantially by an RSI, but there is not 
sufficient supporting data to say that this is a 
generalizable finding either within a state or 
across all states served by one or more of the 
RSIs that were included in this study. 
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Concluding Statements and Needs for the Future 



Embedded in the sections in which the 
findings and statements of impact are 
discussed, one finds some indication of the 
evaluation of the RSI program. In terms of the 
objectives of the evaluation study conducted 
by The Evaluation Center at Western 
Michigan University, the following 
conclusions are appropriate. 

# A system of indicators for each of the 
six drivers was developed through the 
use of a Delphi technique using an 
elite set of professionals (Research 
Advisory Team). 

# The indicators were submitted for 
field validation in the conduct of 15 
case studies and from surveys in 
which their perceived relative 
importance and value were indicated 
by RSI local stakeholders. 

# The status of innovation/reform within 
selected communities was determined 
by on-site case study visits at 15 sites. 
Further, factors that support and those 
that tend to serve as barriers were 
determined as a part of these visits and 
from data derived from interviews, 
focus group meetings, survey data, and 
direct observations. 

# Impact of the RSI experience has been 
determined through analyses of 
achievement test data, surveys, and 
direct observations. The importance 
of the drivers as a measure of this 
impact has been validated and 
reported. 



# We determined that there have been 
some changes in student assessment 
practices, such as direct observations, 
use of written narrative, performance 
testing, etc., but there have been little, 
if any, noticeable changes in the way 
teachers are evaluated as a result of 
RSI participation. 

# An array of factors within local 
schools and communities seem to be 
associated with active involvement 
and systemic reform. These include 
the identification and training of a 
local resource person/coordinator, 
external pressures to align curriculum 
and perform on high stakes testing, 
professional development that 
provides the skills and knowledge to 
align curricula, creating new and 
creative instruction, use of available 
resources to purchase needed 
materials, support for professional 
development, and sustaining initial 
grant-related ventures, etc. The myth 
that rural communities in themselves 
are barriers to reform has not been 
confirmed, and rural communities and 
schools serving rural areas seem as 
willing as any other community to 
accept change and improvement in 
their schools. 

# Partnerships, as represented by the 
large RSI consortia, are fragile and can 
be too large and diverse to be 
effective. One failed largely because 
of multiple jurisdictions, different 
cultures, and the vast area it was 
designed to cover. Early involvement 
in the planning and organization of the 
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partnerships builds ownership and a 
sense that their needs are included. In 
essence, there is a need for common 
interests, a feeling that efforts and 
investments are producing benefits, 
and that students are the benefactors. 
Time is a valuable commodity, and 
that is a paramount concern for 
teachers. 

# Technology, while included in the 
goals and objectives of the RSI 
program and the individual projects, 
is a secondary consideration in 
programming and used very little in 
the administration or coordination of 
the RSIs. One project took on 
technology as a major focus and 



decided at the end of the first year that 
this was a direction that was not 
appropriate or had little potential for 
success. 

Some critical issues from the perspective of 
important stakeholders generally summarize 
their involvement with the RSI program. As a 
part of a survey conducted in 2002 and after 
considerable years of involvement with an 
RSI, school administrator contacts, building 
principals, and local facilitators were asked to 
indicate the three most significant changes, the 
three greatest challenges, and the three most 
important indicators of reform. Their 
responses are summarized on the following 
pages. 
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Summary of Open-End Responses to Critical Issues 



Issues/Questions 

Three Most Significant Changes 

Administrative Contacts School Principals Local Facilitators 

(N = 79) (N = 45) (N = 56) 

Instructional techniques Alignment and improvement of the Use of different and varied 

curriculum instructional strategies and 

assessment techniques 

Science and math emphases Professional development Alignment and changing curricula 

opportunities and results, i.e., 
increased participation, awareness, 
attitudes, and competence 

Teachers’ attitudes, perspectives, Instructional techniques Increasing course requirements 

and awareness 

Teachers’ abilities and involvement Improving course offerings 

in process 

Alignment of curricula Teachers’ attitudes, confidence, 

teamwork, and involvement 

Three Greatest Challenges 

Teacher attitudes, work habits, and Teacher and union attitudes Teacher attitudes and abilities 

involvement 

Teacher qualifications and turnover Time Poor teacher qualifications 

Attitudes of parents and community Parent and community attitudes Time 

Poverty mind-set in the community Lack of community awareness, Inadequate teacher recruitment and 

support, and involvement retention of qualified personnel 

Lack of highly skilled and qualified Work required to obtain training in 

teaching force technology, align curricula, and 

design new lessons 

Lack of opportunity for higher 
education 

Lack of science, math, or 
technology-related jobs in the 
community 

Getting higher education involved 
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Issues/Questions 


Three Most Important Indicators of Reform 


Alignment of curricula to standards 


Inquiry-based lessons with more 
hands-on and laboratory work 


Improved student achievement 


More and more challenging course 
offerings 


New and varied instructional 
practices 


Greater participation of students in 
science and math fairs and 
challenge activities 


Use of the laboratory 


Improved student achievement 


Increased enrollment in more 
challenging STEM courses 


Use of technology 


Improved teacher attitudes 


Student interest and pursuit of 
studies and careers in science, 
math, and technology areas 


Improved teacher attitudes and 
work habits 


More teacher involvement in 
planning and improved work habits 


Percentage of students taking 
qualification exams and enrolling in 
higher education 


Student achievement scores 


Students acceptance of challenges 


Improved student attitudes and 
attendance in school 


Student participation in science 
fairs and other forms of competition 




Greater use of technology 


Student enthusiasm and requests to 
enroll in upper level or more 
challenging courses 




Appearance of the classrooms 






Types of courses offered 






Teacher attitudes and awareness 



While the RSI program was designed with 
lofty goals and anticipation of major changes 
with resulting gains in achievement scores, 
some would conclude that it has fallen short. 
However, one must recognize that (1) the 
approach required by NSF to accomplish these 
outcomes was virtually untested in the 
practical world, especially among schools so 
defined by the RSI regulations; (2) change 
occurs over long periods of time, and with 
changes in tests and testing times, data may 
simply not be reflective of the objectives of 
this program; (3) relatively little monies were 
set aside for local utilization, and no 
commitment for continuing support for 
significance changes was assured; (4) the 



parameters for participation targeted some of 
the most educationally, socially, and 
economically deprived segments of this 
country’s population; and (5) systemic reform 
and improvement in the educational 
experiences in science and math may not 
necessarily be compatible. 

Finally, we discovered conditions and 
situations that are very real in poor, rural 
communities involved in these six RSIs, and 
they are probably present in other areas as 
well. First, the need for improvement in math 
and science education is far greater than we 
assumed, and there are simply not enough 
certified and qualified teachers to meet the 
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needs. Administrators make decisions, maybe 
by necessity or maybe because of outside 
pressures, that possibly meet the “letter of the 
law” but not likely the “intent of the law.” For 
example, the common practice of hiring 
persons with emergency certificates is not in 
the best interests of children, but it does meet 
the requirement that there must be a 
“certified” teacher in each classroom. Teacher 
evaluation procedures are not of the quality 
that differentiates between effective and 
ineffective teachers, and there are instances in 
which continuing employment of a teacher is 
a socioeconomic issue (for him/her and others 
supported by this salary) as opposed to a good 
educational decision. 

In rural communities, such as those we 
studied, there are few employment 
opportunities for high school graduates, much 
less for those with college degrees that require 
an emphasis on math or science. In other 
words, there are very few visible incentives 
for students to pursue secondary or 
postsecondary studies in math or science. 
Strong ties to families prevent many high 
school graduates from leaving the community 
for work or further education. 



Perhaps one of the most prominent 
observations or at least a perception on the 
part of several researchers associated with the 
project is the low level of expectations for 
educational achievement among students, 
teachers, administrators and other community 
members. While they may verbalize strong 
support for education and express their 
interest in students and schools achieving at 
high levels, there seems to be a disconnect 
between this and action. These people are 
poor, and they don’t seem to know how to 
overcome the difficulties of the past or to 
accept the challenges of the future. The status 
quo seems to be an acceptable condition for 
some, and as they see more and more 
repressive education mandates without being 
given the knowledge or resources to respond, 
there likely will be a continually revolving 
door of teachers through the system. Is there 
a need for improved science and math 
education in these rural, poor communities 
and will they accept new and different 
approaches? We think the answer is an 
overwhelming YES. Did the RSI program do 
what it promised or what was hoped? The 
response would have to be NO, but it did 
uncover and help identify the depth and 
breadth of the challenges that lie ahead. 
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